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A basic approach to the problem of evaluating or predicting a crew's 
performance for the VP community is presented. The method uses an ap- 
plication of multiple regression analysis techniques to a model which 
has training parameters as its variables. 

The results would allow the squadron or wing commanding officer to 
predict a crew's performance before the actual flight and to determine 
how to allocate training time for the squadron. 
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I. INTRODUCTION 



A, BACKGROUND 

The patrol squadron ASW (VP) community has been trying to determine 
a method of evaluating crew performance effectively for many years. They 
not only would like to evaluate a performance by a crew but also to pre- 
dict a level of performance by a crew before the flight. The advantages 
of having such a system are obvious. A continuing performance chart of 
each crew, in each squadron would be available. With such a device 
WINGSPAC might be able to do away with yearly Operational Readiness In- 
spections (ORI) . These evaluations have been used to check the ability 
of the squadron to meet any possible contingency. It was given through 
a multitude of flight, written examinations, and ground evolutions. 
Basically it tested the squadrons training program and determined if a 
squadron was ready for deployment. By having these up to date performance 
charts of all crews the Wing could tell at any time, not just at a 
scheduled one (as with the ORI), if that squadron was up to par. 

On this same line the Wing commander could see from these charts if 
a certain squadron 1 s overall performance trend was decreasing. If this 
was accompanied by a heavy period of operational tasking it could mean 
the squadron should be stood down from operational flying so it could 
devote more time to training. The problem of over tasking a squadron 
with operational commitments is a serious problem today facing the Wing 
commander . 

In addition the squadron or Wing commander would be able to determine 
before the flight if a particular crew is qualified to go on that mission 
and do a job of a required degree of skill and competence. 
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B. NATURE OF THE PROBLEM 



At the present time squadrons in WINGSPAC are using a postflight 
evaluation method in an attempt to evaluate a crew’s performance. The 
crew is evaluated by a team of debriefers from the various ASCAC 1 S upon 
completion of the flight. The form is quite lengthy and requires con- 
siderable time to complete during the debrief and after. The range of 
grades goes from 0 to 100 for each area. The number of required areas 
to be graded being dependent on the type of flight flown. 

The major problem here is the standardization of the evaluators in 
how they grade. The variance in their attempts for objectivity in 
quantifying quality has been extremely high. Another area for concern 
is the length of the form. This combined with the large number of 
flights graded and relatively small number of evaluators available causes 
a rushed evaluation with not enough emphasis on constructive criticism. 
Last but not least is the fact that the evaluators are not completely 
devoid of personal connections to the people they are evaluating. 

Because of the above problems the average grade of a sample of 
flights of crews from several squadrons at Moffett Field is the un- 
believably high 96-97% and is attacked by WINGSPAC and the Tactical 
Support Center (TSC) at Moffett Field now. 

The problem that is attacked here is that of predicting a crew’s 
performance. This problem is difficult because of the large number of 
variables affecting performance and the lac of, in some cases, of a 
suitable measure of that variable. 

Because of this it was decided to start by looking strictly at the 
major variables associated with training. The amount and type of train- 
ing might vary between squadrons but it is recognized as being a major 
necessity for all VP crews. 
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A basic approach using multiple regression analysis was then under- 
taken with the selected training variable as independent parameters and 
the postflight evaluation score as the dependent variable. 

The inability of the squadrons to collect data in the required 
format did not allow us to use real world data to test proposed models. 
Data was generated by the author, however, to demonstrate the techniques 
involved. The data used is a reasonable attempt to simulate the actual 
data, based on prior experience. 
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II # METHODOLOGY 



A. THE MODEL 

The independent variables chosen to be looked at are: 

X 2 : Total flight time - the time in hours 

flown as a crew since the crew was formed, 

: Weapons Systems Trainer (WST) time - time 

in hours in trainer as a crew , 

X. : Sub time - time in hours of actual on station 

4 

time working a submarine as a crew , 

The dependent variable is: 

X^ : Postflight evaluation score - overall 

grade of evaluators given as a percent. 

It is expected that the value of X^ i\ 7 ould increase with an increase 
in the independent variable. The rate of increase should decrease as 
the independent variables get larger. One method of mathematically 
describing such a relationship is given by: 



X 1 = 1 - ge"^2 x 2 + a 3 x 2 + °4 X 4^ 



( 1 ) 



where 



0 £ x x £ 1 
0 £ 3 ^ 1 



X 2 , X 3 , x 4 > 0 



Let Z = a 2^2 + a 3^3 a A^4* GC l uat ^ on (1) as the value of Z 

approaches infinity then the value of X^ approaches 1 because the value 

-00 

of e =0. Similarly as the value of Z approaches 0 then the value 
of X 1 approaches (3 because e ^ = 1. 
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This relationship is shown graphically in Figure (1) : 




Figure (1) 
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B. APPLICATION 



Before the model could be used in a linear regression analysis it 
had to be transformed from its non-linear form in equation (1) into a 
linear form. This can be done by using a logarithmic transformation on 
equation (1). Regrouping terms equation (1) becomes 

1 - X l = e ”^2 X 2 + a 3 x 3 + (2) 

Now applying the transformation by taking the natural logarithm of 
both sides equation (2) becomes 

y = In (1 - X L ) = ln3 - Q' 2 X 2 - o? 3 X 3 - a 4 X 4 (3) 

The natural base for the logarithm was chosen for convenience only. 

In this form it is ready to be used in conjuntion with any available 
computer multiple regression package. The author has used the BIMED02R 
stepwise linear regression package [Ref. l] on the IBM 360 computer. 

This program allows variables to enter one at a time and permits the 
user to see in what order and with what effect each does. 

A 

For manual calculations let b be a vector containing the estimators 
of the constant term lnf3, o^, and o^. Let y = lnp . 

A 

v 

A 

a 2 

A 

a 3 

A 

where the symbol ~ indicates that a quantity is an estimator. 



b 
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Let )( bo a matrix of the observations on all independent variables 



augmented by a column of units representing the constant term. 



X = 



1 


X 12 


X 13 


X 14 


1 


X 22 


X 23 


X 24 



# ♦ 



♦ « 



1 X 0 X - X , 
nZ nJ n4 



and x* is \ transpose 



X’ = 



1 


1 


1 


1 


12 


X 22 


X 32 


... x n2 


13 


X 23 


X 33 


* • ’ X n3 


14 


X 24 


X 34 


— X n4 



Let y be a vector of observations of the dependent variable, 

yi 



Y = 



The least squares estimates are given in matrix notation by 

A 

*> = (x'x )" 1 x'Y 



(4) 



If the problem was reduced to only one independent variable, equation 
(3) would simply be 



In (1 - xp = lnp - cr 2 X 2 
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or by some basic substitutions 



y = A + BX 

where A = InP and B = - 

Solving for the estimates of A and B by linear regression the re- 
sults are: 

A A 

A = y - BX 



S (x. - X) (y . - y) 

B - 

n 2 

s (X - X) 

i=l 

where the _ denotes the mean value of all observations of that variable. 

A point estimate of y given a particular value of X can be 
found from the equation 



/\ /S ^ 

y = A + BX" 



where y is the point estimate and X is a particular value of X. 

A 

To use the t-test we must solve for the standard deviation of B. 
This is given by 




The confidence interval for B is 



B ± 



t 



°/2 



n-1 



A 
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C. DATA COLLECTION 



Data has to bo collected by the fleet to test the model. It could 

be done in the following manner. For any flight that is to be graded 

the tactical coordinator of the crew can give the values of the variables 

X 2 , X^ either during the brief or the debrief. After the evaluators 

have finished grading the flight they will record the overall score as 

variable X^ and turn into collection point along with squadron, crew, 

event number, and date. These values of X^, X 2 , X^ 5 and X^ form one 

set of observations. When a reasonable number have been gathered they 

can be used with the regression package and run on the computer or using 

formulas (4) and a calculator the model can be tested and the estimators 

found. Once the estimator of the constant ln(3 and the estimators of the 

coefficients , cv 0 , and a. have been determined these values can be 
15 4 

used to predict future values of the dependent variable X^. 



12 



III. DATA ANALYSIS 



Summary of computer results using data in Table 1 





coefficient 


standard error 


X 2 


-.00105 


.00076 


X 3 


-.00698 


.00272 


X 4 


.00172 


.00994 


constant 


-.97327 




multiple R 


.8859 




R square 


.7848 





A t-test is used to test the hypothesis that the value of o.\ is 
equal to 0. The hypothesis is accepted if 



A 






i 



2,3,4 



Q' 

and rejected if i lies outside this interval. In the avove equation 

O'. 

i 

c is equal to 1- confidence level desired and n-1 is called the degrees 
of freedom, where is the number of observations. 

A t-table can be found in [Refs. 2,3,4, and 5] and many other 
textbooks. Part of a typical table is shown below: 



df. = n-l^^ 


.1 


.05 


.025 


30 


1.310 


1.697 


2.042 


40 


1.303 


1.684 


2.021 


50 


1.298 


1.676 


2.009 



One would enter with the degrees of freedom (df) equal to n-1 down 
the left column and the value of c/2 across the top. 
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For a 907o confidence level for the given data we would enter with 
df ~ 39 and c/2 = .05. The value obtained was 1,685 



-1.684 < - 


.00105 

.00076 


s 1.684 


- 1.684 <; - 


1.381 


<; 1.684 



This tells the user that the coefficient is not significant at the 
90% confidence level or that it is equal to 0 at this level. It can be 
seen from the t-tables that it becomes significant at the 80% level. The 
t value being 1.303. 

a 

Similarly can be tested. Using a 907. confidence level 



- 1.684 £ 



.00698 

.00272 



£ 1.684 



-1.684 <; - 2.564 <; 1.684 



Here Q’^ is significant and different from 0. 

A 

Just by looking at the computer results for a it can be seen that 
the standard error is considerably larger than the coefficient itself. 

A 

This indicates that is not significant at any practical level of con- 
fidence . 

Another important figure in the results is the R square. This in- 
dicates that 78% of the variance in the data is accounted for by the 
model and its significant variables. 

Since a logarithmic transformation was used on the model any term 
in the final results so affected has to be transformed back to its 
usable form in the model. Here only the constant term is affected. 
Taking the antilog of -0.97327 with a natural base the resulting value 
is .3779. 
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The equation for predicting future scores would be 



Xl - 1 - .3779 e -<- 00l05X 2 + -00698X3) 

A A 

With the data used the values of a ' ^ and 0 ^ indicate the relative 

A 

weighing factors of the training parameters and . Since a ^ = ,00105 

A 

and O' ^ = .00698, it would indicate that the training denoted by X^ would 
have approximately 6.5 times the effect on performance as the training 
denoted by X^ . Again this is strictly for the data used in this analysis. 
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IV. CONCLUSIONS 



The approach to the problem is only a basic feasible starting point. 
The model presented may be one of many possible ones. The fleet has to 
take the model presented and test it with actual data collected on a 
higher authority basis, using the methodology presented here. The num- 
erical results found here do not apply to the fleet because they were 
arrived at using generated data. The implications from the tests show 
that the model could work and should be tested. Once this is done the 
estimators that are found can be used to predict the performance of any 
crew on a pending flight. 

An important secondary use of these coefficient estimators is as 
a weighting factor to determine allocation of training time within a 
squadron. The coefficients form a relative merit scale of each parameter 
used in the regression. 

The user can add as many independent variables as he chooses with- 
out complicating the methodology. Since the BIMED02R is a stepwise 
regression model, the user is able to determine the effect of these 
variables individually if he can collect the data in a usable form. 
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APPENDIX A 



DATA USED FOR SAMPLE ANALYSIS 



X 2 


x 3 


231 


20 


20 


8 


30 


8 


38 


11 


54 


11 


64 


14 


830 


152 


842 


152 


850 


152 


878 


155 


902 


158 


453 


45 


461 


48 


490 


51 


498 


54 


513 


57 


240 


23 


252 


26 


261 


26 


270 


29 


136 


20 


492 


86 


316 


72 
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OBS NO. 


x 2 


24 


224 


25 


84 


26 


524 


27 


185 


28 


381 


29 


204 


30 


523 


31 


620 


32 


450 


33 


362 


34 


180 


35 


532 


36 


1226 


37 


239 


38 


781 


39 


796 


40 


93 



X 3 


X 4 


44 


24 


20 


1 


120 


40 


45 


8 


57 


24 


12 


4 


60 


36 


120 


40 


72 


32 


28 


20 


24 


12 


100 


40 


147 


83 


21 


16 


68 


54 


106 


58 


18 


7 
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